Skip to content

fix: parallel_plan and parallel_apply#6427

Closed
nvanheuverzwijn wants to merge 1 commit intorunatlantis:mainfrom
nvanheuverzwijn:patch-2
Closed

fix: parallel_plan and parallel_apply#6427
nvanheuverzwijn wants to merge 1 commit intorunatlantis:mainfrom
nvanheuverzwijn:patch-2

Conversation

@nvanheuverzwijn
Copy link
Copy Markdown
Contributor

what

Fix the parallel plan and apply function.

why

Atlantis should run in parallel.

tests

I tested the change with my test project, which has more than 50 project in it. Without this change, it takes around 15minutes to run the entire thing. With my change, this is significantly reduce to approximately 3minutes, running 15 worker in parallel.

references

closes #6426

Signed-off-by: Nicolas Vanheuverzwijn <nicolas.vanheu@gmail.com>
Copilot AI review requested due to automatic review settings April 24, 2026 13:54
@dosubot dosubot Bot added the go Pull requests that update Go code label Apr 24, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a regression where parallel_plan / parallel_apply effectively serialized execution by removing a workspace-level Git read lock that was held for the full duration of step execution.

Changes:

  • Removed WorkingDir.GitReadLock() acquisition from DefaultProjectCommandRunner.runSteps() to allow multiple project steps to run concurrently within the same PR/workspace.
  • Relies on existing per-project WorkingDirLocker for directory-level mutual exclusion while enabling higher concurrency across projects.

@@ -830,10 +830,6 @@ func (p *DefaultProjectCommandRunner) doStateRm(ctx command.ProjectContext) (out
func (p *DefaultProjectCommandRunner) runSteps(steps []valid.Step, ctx command.ProjectContext, absPath string) ([]string, error) {
var outputs []string

Copy link

Copilot AI Apr 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing the WorkingDir.GitReadLock() around step execution means Clone()/MergeAgain() (which take a git write lock and can reset/merge the workspace) may now run concurrently with terraform steps in the same PR/workspace. That can mutate the working tree mid-plan/apply, leading to inconsistent results or intermittent failures under parallel execution.

Consider keeping a shared read lock while running steps, and instead addressing the parallelism bottleneck by changing WorkingDir.Clone/MergeAgain to avoid taking the write lock when they are effectively no-ops (e.g., acquire a read lock to check whether the workspace is already at the desired ref and only upgrade to a write lock when an update/merge is actually needed), or by cloning/merging once per PR/workspace before starting parallel project workers.

Suggested change
unlock := ctx.WorkingDir.GitReadLock()
defer unlock()

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I beleive we already get a proper lock with

	// Acquire internal lock for the directory we're going to operate in.
	unlockFn, err := p.WorkingDirLocker.TryLock(ctx.Pull.BaseRepo.FullName, ctx.Pull.Num, ctx.Workspace, ctx.RepoRelDir, ctx.ProjectName, command.Plan)
	if err != nil {
		return nil, "", err
	}
	defer unlockFn()	

in doPlan and doApply.

@nvanheuverzwijn nvanheuverzwijn changed the title Update project_command_runner.go fix: parallel_plan and parallel_apply Apr 24, 2026
@nvanheuverzwijn
Copy link
Copy Markdown
Contributor Author

closed in favor of #6376

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

go Pull requests that update Go code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

parallel plan and parallel apply no longer works

2 participants